Building a Scalable Distributed Online Media Processing Environment

Media has become dominant in all aspects of human lives, from critical applications such as medical, military, and security (e.g. surveillance cameras) to entertainment applications such as social media and media sharing websites. Billions of massive media objects (e.g., videos, photos, documents, etc.) are generated every second with high diversity among them (in terms of sizes and formats). These objects have to be stored and retrieved reliably, with low latency and in a scalable while efficient fashion. Additionally, various types of processing are done on media objects, from simple compressions and format conversion, to more complex machine learning algorithms detecting certain patterns and objects. Existing large-scale storage and processing systems face several challenges when handling media objects. My research focuses on building an unified storage and processing environment tailored specifically for media objects, while maintaining high efficiency and scalability. I have built a scalable, load-balanced, efficient storage system optimized for media objects based on their unique access patterns. Currently, I am working on developing an efficient media processing system and integrating these two systems into one framework.